add_exllamav2 #1419
add_exllamav2 #1419
Conversation
| def test_generate_quality(self): | ||
| # don't need to test | ||
| pass | ||
|
|
||
| def test_serialization(self): | ||
| # don't need to test | ||
| pass |
There was a problem hiding this comment.
Here, we quantize the model using the cuda-old kernel and save the model to later load it with exllamav2 for the test_exllama_serialization test. Since these tests will use the cuda-old kernel, we don't need to test them as we already do so in a previous test.
There was a problem hiding this comment.
And how about generate_quality?
There was a problem hiding this comment.
This is also tested in the GPTQTest class. The wording is confusing but test_exllama_serialization in GPTQTestExllamav2 does two things: test the loading quantized weights with exllamav2 kernels + test the inference correctness.
| def test_generate_quality(self): | ||
| # don't need to test | ||
| pass | ||
|
|
||
| def test_serialization(self): | ||
| # don't need to test | ||
| pass |
There was a problem hiding this comment.
And how about generate_quality?
|
I was running some QPeft experiments and it looks like optimum's GPTQ interface does not work with the exllama kernel for finetuning. 
I suspect this is likely due to the absence of the backward function in the exllama kernel. I'm wondering if you guys are aware of this behaviour? |
What does this PR do ?
This PR adds the possibility to choose exllamav2 kernels for GPTQ model This PR follows the integration of the kernels in auto-gptq. I've also added a test to check that we are able to load and infer using exllamav2 kernel. I will update the benchmark in a follow-up PR.